{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# HugeCTR Python Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In HugeCTR version 2.3, we've integrated the Python interface, which supports setting data source and model oversubscribing during training. This notebook explains how to access and use the HugeCTR Python interface. For more details of the usage of Python API, please refer to [HugeCTR Python Interface](../docs/python_interface.md).\n",
    "\n",
    "## Table of Contents\n",
    "1. [Build the HugeCTR Python Interface](#1)\n",
    "1. [Wide & Deep Demo](#2)\n",
    "1. [API Signatures](#3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"1\"></a>\n",
    "## 1. Access the HugeCTR Python Interface\n",
    "\n",
    "1. Please make sure that you have started the notebook inside the running NGC docker container: `nvcr.io/nvidia/merlin/merlin-training:0.5`.\n",
    "\n",
    "   A dynamic link to the `hugectr.so` library is installed to the system path `/usr/local/hugectr/lib/`. Besides, this system path is added to the environment variable `PYTHONPATH`, which means that you can use the Python interface within the docker container environment. Check the dynamic link with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /usr/local/hugectr/lib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Import HugeCTR, in order to train your model with Python as shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hugectr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"2\"></a>\n",
    "## 2. Wide & Deep Demo\n",
    "\n",
    "### 2.1 Download and Preprocess Data\n",
    "1. Go to [this link](https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/), and download one of 24 files into the directory `${project_root}/tools`, or execute the following command:\n",
    "   ```\n",
    "   $ cd ${project_root}/tools\n",
    "   $ wget http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_1.gz\n",
    "   ```\n",
    "2. Extract the dataset using the following command:\n",
    "   ```shell\n",
    "   $ tar zxvf day_1.gz\n",
    "   ```\n",
    "\n",
    "3. Preprocess the data using `tools/preprocess.sh`:\n",
    "   ```shell\n",
    "   $ bash tools/preprocess.sh 1 wdl_data pandas 1 1 100 # using pandas to preprocess the data\n",
    "   ```\n",
    "   The command will run about half an hour, finally generate a folder named wdl_data containing preprocessed data. For more detailed usage of `tools/preprocess.sh`, please refer the #Preprocessing by Pandas# in [samples/wdl](../samples/wdl/README.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Train from scratch\n",
    "\n",
    "We can train fom scratch and store the trained dense model and embedding tables in model files by doing the following: \n",
    "\n",
    "1. Create a JSON file for the W&D model. \n",
    "   **NOTE**: Please note that the solver clause no longer needs to be added to the JSON file when using the Python interface. Instead, you can configure the parameters using `hugectr.solver_parser_helper()` directly in the Python interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing wdl_1gpu.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile wdl_1gpu.json\n",
    "{\n",
    "  \"optimizer\": {\n",
    "    \"type\": \"Adam\",\n",
    "    \"update_type\": \"Global\",\n",
    "    \"adam_hparam\": {\n",
    "      \"learning_rate\": 0.001,\n",
    "      \"beta1\": 0.9,\n",
    "      \"beta2\": 0.999,\n",
    "      \"epsilon\": 0.0000001\n",
    "    }\n",
    "  },\n",
    "  \"layers\": [\n",
    "    {\n",
    "      \"name\": \"data\",\n",
    "      \"type\": \"Data\",\n",
    "      \"source\": \"wdl_data/file_list.0.txt\",\n",
    "      \"eval_source\": \"wdl_data/file_list.8.txt\",\n",
    "      \"check\": \"Sum\",\n",
    "      \"label\": {\n",
    "        \"top\": \"label\",\n",
    "        \"label_dim\": 1\n",
    "      },\n",
    "      \"dense\": {\n",
    "        \"top\": \"dense\",\n",
    "        \"dense_dim\": 13\n",
    "      },\n",
    "      \"sparse\": [\n",
    "        {\n",
    "          \"top\": \"wide_data\",\n",
    "          \"type\": \"DistributedSlot\",\n",
    "          \"max_feature_num_per_sample\": 30,\n",
    "          \"slot_num\": 1\n",
    "        },\n",
    "        {\n",
    "          \"top\": \"deep_data\",\n",
    "          \"type\": \"DistributedSlot\",\n",
    "          \"max_feature_num_per_sample\": 30,\n",
    "          \"slot_num\": 26\n",
    "        }\n",
    "      ]\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"sparse_embedding2\",\n",
    "      \"type\": \"DistributedSlotSparseEmbeddingHash\",\n",
    "      \"bottom\": \"wide_data\",\n",
    "      \"top\": \"sparse_embedding2\",\n",
    "      \"sparse_embedding_hparam\": {\n",
    "        \"max_vocabulary_size_per_gpu\": 5863985,\n",
    "        \"embedding_vec_size\": 1,\n",
    "        \"combiner\": 0\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"sparse_embedding1\",\n",
    "      \"type\": \"DistributedSlotSparseEmbeddingHash\",\n",
    "      \"bottom\": \"deep_data\",\n",
    "      \"top\": \"sparse_embedding1\",\n",
    "      \"sparse_embedding_hparam\": {\n",
    "        \"max_vocabulary_size_per_gpu\": 5863985,\n",
    "        \"embedding_vec_size\": 16,\n",
    "        \"combiner\": 0\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"reshape1\",\n",
    "      \"type\": \"Reshape\",\n",
    "      \"bottom\": \"sparse_embedding1\",\n",
    "      \"top\": \"reshape1\",\n",
    "      \"leading_dim\": 416\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"reshape2\",\n",
    "      \"type\": \"Reshape\",\n",
    "      \"bottom\": \"sparse_embedding2\",\n",
    "      \"top\": \"reshape2\",\n",
    "      \"leading_dim\": 1\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"concat1\",\n",
    "      \"type\": \"Concat\",\n",
    "      \"bottom\": [\n",
    "        \"reshape1\",\n",
    "        \"dense\"\n",
    "      ],\n",
    "      \"top\": \"concat1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc1\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"concat1\",\n",
    "      \"top\": \"fc1\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1024\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"relu1\",\n",
    "      \"type\": \"ReLU\",\n",
    "      \"bottom\": \"fc1\",\n",
    "      \"top\": \"relu1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"dropout1\",\n",
    "      \"type\": \"Dropout\",\n",
    "      \"rate\": 0.5,\n",
    "      \"bottom\": \"relu1\",\n",
    "      \"top\": \"dropout1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc2\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"dropout1\",\n",
    "      \"top\": \"fc2\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1024\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"relu2\",\n",
    "      \"type\": \"ReLU\",\n",
    "      \"bottom\": \"fc2\",\n",
    "      \"top\": \"relu2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"dropout2\",\n",
    "      \"type\": \"Dropout\",\n",
    "      \"rate\": 0.5,\n",
    "      \"bottom\": \"relu2\",\n",
    "      \"top\": \"dropout2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc4\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"dropout2\",\n",
    "      \"top\": \"fc4\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"add1\",\n",
    "      \"type\": \"Add\",\n",
    "      \"bottom\": [\n",
    "        \"fc4\",\n",
    "        \"reshape2\"\n",
    "      ],\n",
    "      \"top\": \"add1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"loss\",\n",
    "      \"type\": \"BinaryCrossEntropyLoss\",\n",
    "      \"bottom\": [\n",
    "        \"add1\",\n",
    "        \"label\"\n",
    "      ],\n",
    "      \"top\": \"loss\"\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Write the Python script. \n",
    "   Ensure that the `repeat_dataset` parameter is set to `False` within the script, which indicates that the file list needs to be specified before submitting the sess.train() or sess.evaluation() calls. Additionally, be sure to create a write-enabled directory for storing the temporary files for model oversubscribing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing wdl_from_scratch.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile wdl_from_scratch.py\n",
    "from hugectr import Session, solver_parser_helper\n",
    "import sys\n",
    "from mpi4py import MPI\n",
    "\n",
    "def train_from_scratch(json_file):\n",
    "  dataset = [(\"./wdl_data/file_list.\"+str(i)+\".txt\", \"./wdl_data/file_list.\"+str(i)+\".keyset\") for i in range(8)]\n",
    "  solver_config = solver_parser_helper(seed = 0,\n",
    "                                     batchsize = 16384,\n",
    "                                     batchsize_eval =16384,\n",
    "                                     model_file = \"\",\n",
    "                                     embedding_files = [],\n",
    "                                     vvgpu = [[0]],\n",
    "                                     use_mixed_precision = False,\n",
    "                                     scaler = 1.0,\n",
    "                                     i64_input_key = False,\n",
    "                                     use_algorithm_search = True,\n",
    "                                     use_cuda_graph = True,\n",
    "                                     repeat_dataset = False\n",
    "                                    )\n",
    "  sess = Session(solver_config, json_file, True, \"./temp_embedding\")\n",
    "  data_reader_train = sess.get_data_reader_train()\n",
    "  data_reader_eval = sess.get_data_reader_eval()\n",
    "  data_reader_eval.set_source(\"./wdl_data/file_list.8.txt\")\n",
    "  model_oversubscriber = sess.get_model_oversubscriber()\n",
    "  iteration = 0\n",
    "  for file_list, keyset_file in dataset:\n",
    "    data_reader_train.set_source(file_list)\n",
    "    model_oversubscriber.update(keyset_file)\n",
    "    while True:\n",
    "      good = sess.train()\n",
    "      if good == False:\n",
    "        break\n",
    "      if iteration % 100 == 0:\n",
    "        sess.check_overflow()\n",
    "        sess.copy_weights_for_evaluation()\n",
    "        data_reader_eval = sess.get_data_reader_eval()\n",
    "        good_eval = True\n",
    "        j = 0\n",
    "        while good_eval:\n",
    "          if j >= solver_config.max_eval_batches:\n",
    "            break\n",
    "          good_eval = sess.eval()\n",
    "          j += 1\n",
    "        if good_eval == False:\n",
    "          data_reader_eval.set_source()\n",
    "        metrics = sess.get_eval_metrics()\n",
    "        print(\"[HUGECTR][INFO] iter: {}, metrics: {}\".format(iteration, metrics))\n",
    "      iteration += 1\n",
    "    print(\"[HUGECTR][INFO] trained with data in {}\".format(file_list))\n",
    "  sess.download_params_to_files(\"./\", iteration)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "  json_file = sys.argv[1]\n",
    "  train_from_scratch(json_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing wdl_from_scratch.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile wdl_from_scratch.sh\n",
    "mkdir -p temp_embedding && \\\n",
    "python3 wdl_from_scratch.py wdl_1gpu.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[02d01h44m32s][HUGECTR][INFO]: Global seed is 3078712038\n",
      "[02d01h44m35s][HUGECTR][INFO]: Peer-to-peer access cannot be fully enabled.\n",
      "Device 0: GeForce RTX 2080 Ti\n",
      "[02d01h44m35s][HUGECTR][INFO]: cache_eval_data is not specified using default: 0\n",
      "[02d01h44m35s][HUGECTR][INFO]: num_workers is not specified using default: 12\n",
      "[02d01h44m35s][HUGECTR][INFO]: num of DataReader workers: 12\n",
      "[02d01h44m35s][HUGECTR][INFO]: max_nnz is not specified using default: 30\n",
      "[02d01h44m35s][HUGECTR][INFO]: max_nnz is not specified using default: 30\n",
      "[02d01h44m35s][HUGECTR][INFO]: num_internal_buffers 1\n",
      "[02d01h44m35s][HUGECTR][INFO]: num_internal_buffers 1\n",
      "[02d01h44m35s][HUGECTR][INFO]: max_vocabulary_size_per_gpu_=5863985\n",
      "[02d01h44m35s][HUGECTR][INFO]: max_vocabulary_size_per_gpu_=5863985\n",
      "[02d01h45m16s][HUGECTR][INFO]: Traning from scratch, no snapshot file specified\n",
      "[02d01h45m16s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h45m16s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h45m17s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 0\n",
      "[02d01h45m17s][HUGECTR][INFO]: Done\n",
      "[02d01h45m17s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 0\n",
      "[02d01h45m17s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 0, metrics: [('AUC', 0.4978078603744507)]\n",
      "[HUGECTR][INFO] iter: 100, metrics: [('AUC', 0.7422940731048584)]\n",
      "[HUGECTR][INFO] iter: 200, metrics: [('AUC', 0.7539043426513672)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.0.txt\n",
      "[02d01h46m01s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h46m01s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h46m02s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h46m02s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h46m03s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 442\n",
      "[02d01h46m03s][HUGECTR][INFO]: Done\n",
      "[02d01h46m04s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 820\n",
      "[02d01h46m04s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 300, metrics: [('AUC', 0.7544599771499634)]\n",
      "[HUGECTR][INFO] iter: 400, metrics: [('AUC', 0.760477602481842)]\n",
      "[HUGECTR][INFO] iter: 500, metrics: [('AUC', 0.7647740244865417)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.1.txt\n",
      "[02d01h46m27s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h46m27s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h46m27s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h46m27s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h46m29s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 555\n",
      "[02d01h46m29s][HUGECTR][INFO]: Done\n",
      "[02d01h46m30s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 960\n",
      "[02d01h46m30s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 600, metrics: [('AUC', 0.7630816698074341)]\n",
      "[HUGECTR][INFO] iter: 700, metrics: [('AUC', 0.7671055793762207)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.2.txt\n",
      "[02d01h46m54s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h46m54s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h46m54s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h46m54s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h46m56s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 619\n",
      "[02d01h46m56s][HUGECTR][INFO]: Done\n",
      "[02d01h46m58s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1014\n",
      "[02d01h46m58s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 800, metrics: [('AUC', 0.7653575539588928)]\n",
      "[HUGECTR][INFO] iter: 900, metrics: [('AUC', 0.7663018703460693)]\n",
      "[HUGECTR][INFO] iter: 1000, metrics: [('AUC', 0.7696205973625183)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.3.txt\n",
      "[02d01h47m21s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h47m21s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h47m22s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h47m22s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h47m24s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 660\n",
      "[02d01h47m24s][HUGECTR][INFO]: Done\n",
      "[02d01h47m25s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1036\n",
      "[02d01h47m25s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 1100, metrics: [('AUC', 0.7684949636459351)]\n",
      "[HUGECTR][INFO] iter: 1200, metrics: [('AUC', 0.7705279588699341)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.4.txt\n",
      "[02d01h47m48s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h47m48s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h47m49s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h47m49s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h47m50s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 690\n",
      "[02d01h47m51s][HUGECTR][INFO]: Done\n",
      "[02d01h47m52s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1048\n",
      "[02d01h47m52s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 1300, metrics: [('AUC', 0.7694478631019592)]\n",
      "[HUGECTR][INFO] iter: 1400, metrics: [('AUC', 0.7711092233657837)]\n",
      "[HUGECTR][INFO] iter: 1500, metrics: [('AUC', 0.7721576690673828)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.5.txt\n",
      "[02d01h48m15s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h48m15s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h48m15s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h48m15s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h48m17s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 714\n",
      "[02d01h48m17s][HUGECTR][INFO]: Done\n",
      "[02d01h48m19s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1052\n",
      "[02d01h48m19s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 1600, metrics: [('AUC', 0.7716618776321411)]\n",
      "[HUGECTR][INFO] iter: 1700, metrics: [('AUC', 0.7727320790290833)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.6.txt\n",
      "[02d01h48m42s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h48m42s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h48m42s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[02d01h48m42s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[02d01h48m44s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 733\n",
      "[02d01h48m44s][HUGECTR][INFO]: Done\n",
      "[02d01h48m46s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1055\n",
      "[02d01h48m46s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 1800, metrics: [('AUC', 0.771714985370636)]\n",
      "[HUGECTR][INFO] iter: 1900, metrics: [('AUC', 0.7732033729553223)]\n",
      "[HUGECTR][INFO] iter: 2000, metrics: [('AUC', 0.773735761642456)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.7.txt\n",
      "[02d01h49m09s][HUGECTR][INFO]: Rank0: Dump hash table from GPU0\n",
      "[02d01h49m09s][HUGECTR][INFO]: Rank0: Write hash table <key,value> pairs to file\n",
      "[02d01h49m09s][HUGECTR][INFO]: Done\n",
      "[02d01h49m09s][HUGECTR][INFO]: Rank0: Dump hash table from GPU0\n",
      "[02d01h49m09s][HUGECTR][INFO]: Rank0: Write hash table <key,value> pairs to file\n",
      "[02d01h49m09s][HUGECTR][INFO]: Done\n"
     ]
    }
   ],
   "source": [
    "!bash wdl_from_scratch.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Train from stored model\n",
    "\n",
    "Check the stored model files that will be used in the training. Dense model file embeddings should be passed to the respective model_file and embedding_files when calling `sess.solver_parser_helper()`. We will use the same JSON file and training data as the previous section. Also, all the other configurations for `solver_parser_helper` will also be the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0_sparse_2016.model  1_sparse_2016.model  _dense_2016.model\r\n"
     ]
    }
   ],
   "source": [
    "!ls *.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting wdl_from_stored.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile wdl_from_stored.py\n",
    "from hugectr import Session, solver_parser_helper\n",
    "import sys\n",
    "from mpi4py import MPI\n",
    "\n",
    "def train_from_stored(json_file):\n",
    "  dataset = [(\"./wdl_data/file_list.\"+str(i)+\".txt\", \"./wdl_data/file_list.\"+str(i)+\".keyset\") for i in range(8)]\n",
    "  solver_config = solver_parser_helper(seed = 0,\n",
    "                                     batchsize = 16384,\n",
    "                                     batchsize_eval =16384,\n",
    "                                     model_file = \"_dense_2016.model\",\n",
    "                                     embedding_files = [\"0_sparse_2016.model\", \"1_sparse_2016.model\"],\n",
    "                                     vvgpu = [[0]],\n",
    "                                     use_mixed_precision = False,\n",
    "                                     scaler = 1.0,\n",
    "                                     i64_input_key = False,\n",
    "                                     use_algorithm_search = True,\n",
    "                                     use_cuda_graph = True,\n",
    "                                     repeat_dataset = False\n",
    "                                    )\n",
    "  sess = Session(solver_config, json_file, True, \"./temp_embedding\")\n",
    "  data_reader_train = sess.get_data_reader_train()\n",
    "  data_reader_eval = sess.get_data_reader_eval()\n",
    "  data_reader_eval.set_source(\"./wdl_data/file_list.8.txt\")\n",
    "  model_oversubscriber = sess.get_model_oversubscriber()\n",
    "  iteration = 1260\n",
    "  for file_list, keyset_file in dataset:\n",
    "    data_reader_train.set_source(file_list)\n",
    "    model_oversubscriber.update(keyset_file)\n",
    "    while True:\n",
    "      good = sess.train()\n",
    "      if good == False:\n",
    "        break\n",
    "      if iteration % 100 == 0:\n",
    "        sess.check_overflow()\n",
    "        sess.copy_weights_for_evaluation()\n",
    "        data_reader_eval = sess.get_data_reader_eval()\n",
    "        good_eval = True\n",
    "        j = 0\n",
    "        while good_eval:\n",
    "          if j >= solver_config.max_eval_batches:\n",
    "            break\n",
    "          good_eval = sess.eval()\n",
    "          j += 1\n",
    "        if good_eval == False:\n",
    "          data_reader_eval.set_source()\n",
    "        metrics = sess.get_eval_metrics()\n",
    "        print(\"[HUGECTR][INFO] iter: {}, metrics: {}\".format(iteration, metrics))\n",
    "      iteration += 1\n",
    "    print(\"[HUGECTR][INFO] trained with data in {}\".format(file_list))\n",
    "  sess.download_params_to_files(\"./\", iteration)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "  json_file = sys.argv[1]\n",
    "  train_from_stored(json_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing wdl_from_stored.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile wdl_from_stored.sh\n",
    "mkdir -p temp_embedding && \\\n",
    "python3 wdl_from_stored.py wdl_1gpu.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[01d13h17m31s][HUGECTR][INFO]: Global seed is 431843434\n",
      "[01d13h17m32s][HUGECTR][INFO]: Peer-to-peer access cannot be fully enabled.\n",
      "Device 0: GeForce RTX 2080 Ti\n",
      "[01d13h17m32s][HUGECTR][INFO]: cache_eval_data is not specified using default: 0\n",
      "[01d13h17m32s][HUGECTR][INFO]: num_workers is not specified using default: 12\n",
      "[01d13h17m32s][HUGECTR][INFO]: num of DataReader workers: 12\n",
      "[01d13h17m32s][HUGECTR][INFO]: max_nnz is not specified using default: 30\n",
      "[01d13h17m32s][HUGECTR][INFO]: max_nnz is not specified using default: 30\n",
      "[01d13h17m32s][HUGECTR][INFO]: num_internal_buffers 1\n",
      "[01d13h17m32s][HUGECTR][INFO]: num_internal_buffers 1\n",
      "[01d13h17m37s][HUGECTR][INFO]: max_vocabulary_size_per_gpu_=5863985\n",
      "[01d13h17m37s][HUGECTR][INFO]: max_vocabulary_size_per_gpu_=5863985\n",
      "Loading dense model: _dense_2016.model\n",
      "[01d13h18m24s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h18m24s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h18m25s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 442\n",
      "[01d13h18m25s][HUGECTR][INFO]: Done\n",
      "[01d13h18m26s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 821\n",
      "[01d13h18m26s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 1300, metrics: [('AUC', 0.7719355821609497)]\n",
      "[HUGECTR][INFO] iter: 1400, metrics: [('AUC', 0.7717985510826111)]\n",
      "[HUGECTR][INFO] iter: 1500, metrics: [('AUC', 0.7717432379722595)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.0.txt\n",
      "[01d13h24m29s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h24m29s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h24m30s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h24m30s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h24m32s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 555\n",
      "[01d13h24m32s][HUGECTR][INFO]: Done\n",
      "[01d13h24m33s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 959\n",
      "[01d13h24m33s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 1600, metrics: [('AUC', 0.7691575884819031)]\n",
      "[HUGECTR][INFO] iter: 1700, metrics: [('AUC', 0.7704731225967407)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.1.txt\n",
      "[01d13h32m47s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h32m47s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h32m48s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h32m48s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h32m50s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 618\n",
      "[01d13h32m50s][HUGECTR][INFO]: Done\n",
      "[01d13h32m51s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1014\n",
      "[01d13h32m51s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 1800, metrics: [('AUC', 0.7660712599754333)]\n",
      "[HUGECTR][INFO] iter: 1900, metrics: [('AUC', 0.7713440656661987)]\n",
      "[HUGECTR][INFO] iter: 2000, metrics: [('AUC', 0.7716435790061951)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.2.txt\n",
      "[01d13h40m28s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h40m28s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h40m29s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h40m29s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h40m33s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 662\n",
      "[01d13h40m33s][HUGECTR][INFO]: Done\n",
      "[01d13h40m35s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1038\n",
      "[01d13h40m35s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 2100, metrics: [('AUC', 0.7696810960769653)]\n",
      "[HUGECTR][INFO] iter: 2200, metrics: [('AUC', 0.7733792662620544)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.3.txt\n",
      "[01d13h46m58s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h46m58s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h46m59s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h46m59s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h47m02s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 691\n",
      "[01d13h47m02s][HUGECTR][INFO]: Done\n",
      "[01d13h47m03s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1047\n",
      "[01d13h47m03s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 2300, metrics: [('AUC', 0.766020655632019)]\n",
      "[HUGECTR][INFO] iter: 2400, metrics: [('AUC', 0.7715461254119873)]\n",
      "[HUGECTR][INFO] iter: 2500, metrics: [('AUC', 0.7737528681755066)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.4.txt\n",
      "[01d13h50m45s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h50m45s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h50m46s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h50m46s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h50m48s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 714\n",
      "[01d13h50m48s][HUGECTR][INFO]: Done\n",
      "[01d13h50m50s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1052\n",
      "[01d13h50m50s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 2600, metrics: [('AUC', 0.7713172435760498)]\n",
      "[HUGECTR][INFO] iter: 2700, metrics: [('AUC', 0.7725634574890137)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.5.txt\n",
      "[01d13h54m46s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h54m46s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h54m46s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h54m46s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h54m49s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 732\n",
      "[01d13h54m49s][HUGECTR][INFO]: Done\n",
      "[01d13h54m50s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1053\n",
      "[01d13h54m50s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 2800, metrics: [('AUC', 0.7703032493591309)]\n",
      "[HUGECTR][INFO] iter: 2900, metrics: [('AUC', 0.7738139033317566)]\n",
      "[HUGECTR][INFO] iter: 3000, metrics: [('AUC', 0.7731029391288757)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.6.txt\n",
      "[01d13h58m38s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h58m38s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h58m39s][HUGECTR][INFO]: Dump hash table from GPU0\n",
      "[01d13h58m39s][HUGECTR][INFO]: Write hash table <key,value> pairs to file\n",
      "[01d13h58m42s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1006\n",
      "[01d13h58m42s][HUGECTR][INFO]: Done\n",
      "[01d13h58m44s][HUGECTR][INFO]: Start to upload embedding table file to GPUs, total loop_num: 1055\n",
      "[01d13h58m44s][HUGECTR][INFO]: Done\n",
      "[HUGECTR][INFO] iter: 3100, metrics: [('AUC', 0.7722951769828796)]\n",
      "[HUGECTR][INFO] iter: 3200, metrics: [('AUC', 0.7728274464607239)]\n",
      "[HUGECTR][INFO] trained with data in ./wdl_data/file_list.7.txt\n",
      "[01d14h03m34s][HUGECTR][INFO]: Rank0: Dump hash table from GPU0\n",
      "[01d14h03m34s][HUGECTR][INFO]: Rank0: Write hash table <key,value> pairs to file\n",
      "[01d14h03m34s][HUGECTR][INFO]: Done\n",
      "[01d14h03m34s][HUGECTR][INFO]: Rank0: Dump hash table from GPU0\n",
      "[01d14h03m34s][HUGECTR][INFO]: Rank0: Write hash table <key,value> pairs to file\n",
      "[01d14h03m34s][HUGECTR][INFO]: Done\n"
     ]
    }
   ],
   "source": [
    "!bash wdl_from_stored.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"3\"></a>\n",
    "## 3. API Signatures\n",
    "\n",
    "Here's list of all the API signatures within the HugeCTR Python interface that you need to get familiar with to successfully train your own model. As you can see from the above example, we've included `Session`, `DataReader`, `ModelPrefetcher` and `solver_parser_helper`.\n",
    "\n",
    "**Session**\n",
    "```bash\n",
    "class Session(pybind11_builtins.pybind11_object)\n",
    " |  Method resolution order:\n",
    " |      Session\n",
    " |      pybind11_builtins.pybind11_object\n",
    " |      builtins.object\n",
    " |\n",
    " |  Methods defined here:\n",
    " |\n",
    " |  check_overflow(...)\n",
    " |      check_overflow(self: hugectr.Session) -> None\n",
    " |\n",
    " |  copy_weights_for_evaluation(...)\n",
    " |      copy_weights_for_evaluation(self: hugectr.Session) -> None\n",
    " |\n",
    " |  download_params_to_files(...)\n",
    " |      download_params_to_files(self: hugectr.Session, prefix: str, iter: int) -> hugectr\n",
    ".Error_t\n",
    " |\n",
    " |  eval(...)\n",
    " |      eval(self: hugectr.Session) -> bool\n",
    " |\n",
    " |  get_current_loss(...)\n",
    " |      get_current_loss(self: hugectr.Session) -> float\n",
    " |\n",
    " |  get_model_oversubscriber(...)\n",
    " |      get_model_oversubscriber(self: hugectr.Session) -> hugectr.ModelOversubscriber\n",
    " |\n",
    " |  get_params_num(...)\n",
    " |      get_params_num(self: hugectr.Session) -> int\n",
    " |\n",
    " |  init_params(...)\n",
    " |      init_params(self: hugectr.Session, model_file: str) -> hugectr.Error_t\n",
    " |\n",
    " |  set_learning_rate(...)\n",
    " |      set_learning_rate(self: hugectr.Session, lr: float) -> hugectr.Error_t\n",
    " |\n",
    " |  start_data_reading(...)\n",
    " |      start_data_reading(self: hugectr.Session) -> None\n",
    " |\n",
    " |  train(...)\n",
    " |      train(self: hugectr.Session) -> bool\n",
    "```\n",
    "\n",
    "**DataReader**\n",
    "```bash\n",
    "class DataReader32(IDataReader)\n",
    " |  Method resolution order:\n",
    " |      DataReader32\n",
    " |      IDataReader\n",
    " |      pybind11_builtins.pybind11_object\n",
    " |      builtins.object\n",
    " |\n",
    " |  Methods defined here:\n",
    " |\n",
    " |  __init__(...)\n",
    " |      __init__(self: hugectr.DataReader32, batchsize: int, label_dim: int, dense_dim: in\n",
    "t, params: List[hugectr.DataReaderSparseParam], resource_manager: hugectr.ResourceManager,\n",
    " repeat: bool, num_chunk_threads: int, use_mixed_precision: bool, cache_num_iters: int) ->\n",
    " None\n",
    " |\n",
    " |  create_drwg_norm(...)\n",
    " |      create_drwg_norm(self: hugectr.DataReader32, file_list: str, Check_t: hugectr.Chec\n",
    "k_t, start_reading_from_beginning: bool=True) -> None\n",
    " |\n",
    " |  create_drwg_parquet(...)\n",
    " |      create_drwg_parquet(self: hugectr.DataReader32, file_list: str, slot_offset: List[\n",
    "int], start_reading_from_beginning: bool=True) -> None\n",
    " |\n",
    " |  create_drwg_raw(...)\n",
    " |      create_drwg_raw(self: hugectr.DataReader32, file_name: str, num_samples: int, slot\n",
    "_offset: List[int], float_label_dense: bool, data_shuffle: bool=False, start_reading_from_\n",
    "beginning: bool=True) -> None\n",
    " |\n",
    " |  get_dense_tensors(...)\n",
    " |      get_dense_tensors(self: hugectr.DataReader32) -> List[HugeCTR::TensorBag2]\n",
    " |\n",
    " |  get_label_tensors(...)\n",
    " |      get_label_tensors(self: hugectr.DataReader32) -> List[HugeCTR::Tensor2<float>]\n",
    " |\n",
    " |  get_row_offsets_tensors(...)\n",
    " |      get_row_offsets_tensors(*args, **kwargs)\n",
    " |      Overloaded function.\n",
    " |\n",
    " |      1. get_row_offsets_tensors(self: hugectr.DataReader32) -> List[HugeCTR::Tensor2<un\n",
    "signed int>]\n",
    " |\n",
    " |      2. get_value_tensors(self: hugectr.DataReader32, param_id: int) -> List[HugeCTR::T\n",
    "ensor2<unsigned int>]\n",
    " |\n",
    " |  read_a_batch_to_device(...)\n",
    " |      read_a_batch_to_device(self: hugectr.DataReader32) -> int\n",
    " |\n",
    " |  read_a_batch_to_device_delay_release(...)\n",
    " |      read_a_batch_to_device_delay_release(self: hugectr.DataReader32) -> int\n",
    " |\n",
    " |  ready_to_collect(...)\n",
    " |      ready_to_collect(self: hugectr.DataReader32) -> None\n",
    " |\n",
    " |  set_source(...)\n",
    " |      set_source(self: hugectr.DataReader32, file_name: str='') -> None\n",
    " |\n",
    " |  start(...)\n",
    " |      start(self: hugectr.DataReader32) -> None\n",
    "\n",
    "class DataReader64(IDataReader)\n",
    " |  Method resolution order:\n",
    " |      DataReader64\n",
    " |      IDataReader\n",
    " |      pybind11_builtins.pybind11_object\n",
    " |      builtins.object\n",
    " |\n",
    " |  Methods defined here:\n",
    " |\n",
    " |  create_drwg_norm(...)\n",
    " |      create_drwg_norm(self: hugectr.DataReader64, file_list: str, Check_t: hugectr.Chec\n",
    "k_t, start_reading_from_beginning: bool=True) -> None\n",
    " |\n",
    " |  create_drwg_parquet(...)\n",
    " |      create_drwg_parquet(self: hugectr.DataReader64, file_list: str, slot_offset: List[\n",
    "int], start_reading_from_beginning: bool=True) -> None\n",
    " |\n",
    " |  create_drwg_raw(...)\n",
    " |      create_drwg_raw(self: hugectr.DataReader64, file_name: str, num_samples: int, slot\n",
    "_offset: List[int], float_label_dense: bool, data_shuffle: bool=False, start_reading_from_\n",
    "beginning: bool=True) -> None\n",
    " |\n",
    " |  get_dense_tensors(...)\n",
    " |      get_dense_tensors(self: hugectr.DataReader64) -> List[HugeCTR::TensorBag2]\n",
    " |\n",
    " |  get_label_tensors(...)\n",
    " |      get_label_tensors(self: hugectr.DataReader64) -> List[HugeCTR::Tensor2<float>]\n",
    " |\n",
    " |  get_row_offsets_tensors(...)\n",
    " |      get_row_offsets_tensors(*args, **kwargs)\n",
    " |      Overloaded function.\n",
    " |\n",
    " |      1. get_row_offsets_tensors(self: hugectr.DataReader64) -> List[HugeCTR::Tensor2<lo\n",
    "ng long>]\n",
    " |\n",
    " |      2. get_row_offsets_tensors(self: hugectr.DataReader64, param_id: int) -> List[Huge\n",
    "CTR::Tensor2<long long>]\n",
    " |\n",
    " |  get_value_tensors(...)\n",
    " |      get_value_tensors(*args, **kwargs)\n",
    " |      Overloaded function.\n",
    " |\n",
    " |      1. get_value_tensors(self: hugectr.DataReader64) -> List[HugeCTR::Tensor2<long lon\n",
    "g>]\n",
    " |\n",
    " |      2. get_value_tensors(self: hugectr.DataReader64, param_id: int) -> List[HugeCTR::T\n",
    "ensor2<long long>]\n",
    " |\n",
    " |  read_a_batch_to_device(...)\n",
    " |      read_a_batch_to_device(self: hugectr.DataReader64) -> int\n",
    " |\n",
    " |  read_a_batch_to_device_delay_release(...)\n",
    " |      read_a_batch_to_device_delay_release(self: hugectr.DataReader64) -> int\n",
    " |\n",
    " |  ready_to_collect(...)\n",
    " |      ready_to_collect(self: hugectr.DataReader64) -> None\n",
    " |\n",
    " |  set_source(...)\n",
    " |      set_source(self: hugectr.DataReader64, file_name: str='') -> None\n",
    " |\n",
    " |  start(...)\n",
    " |      start(self: hugectr.DataReader64) -> None\n",
    "```\n",
    "\n",
    "**ModelPrefetcher**\n",
    "```bash\n",
    "class ModelOversubscriber(pybind11_builtins.pybind11_object)\n",
    " |  Method resolution order:\n",
    " |      ModelOversubscriber\n",
    " |      pybind11_builtins.pybind11_object\n",
    " |      builtins.object\n",
    " |\n",
    " |  Methods defined here:\n",
    " |\n",
    " |  store(...)\n",
    " |      store(self: hugectr.ModelOversubscriber, snapshot_file_list: List[str]) -> None\n",
    " |\n",
    " |  update(...)\n",
    " |      update(*args, **kwargs)\n",
    " |      Overloaded function.\n",
    " |\n",
    " |      1. update(self: hugectr.ModelOversubscriber, keyset_file: str) -> None\n",
    " |\n",
    " |      2. update(self: hugectr.ModelOversubscriber, keyset_file_list: List[str]) -> None\n",
    " ```\n",
    " \n",
    " **solver_parser_helper**\n",
    " ```bash\n",
    " solver_parser_helper(...) method of builtins.PyCapsule instance\n",
    "    solver_parser_helper(seed: int=0, max_eval_batches: int=100, batchsize_eval: int=2048,\n",
    " batchsize: int=2048, model_file: str='', embedding_files: List[str]=[], vvgpu: List[List[\n",
    "int]]=[[0]], use_mixed_precision: bool=False, enable_tf32_compute: bool=False, scaler: flo\n",
    "at=1.0, i64_input_key: bool=False, use_algorithm_search: bool=True, use_cuda_graph: bool=T\n",
    "rue, repeat_dataset: bool=True, max_iter: int=0, num_epochs: int=0, display: int=200, snap\n",
    "shot: int=10000, eval_interval: int=1000, use_model_oversubscriber: bool=False, temp_embed\n",
    "ding_dir: str='./') -> hugectr.SolverParser\n",
    " ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
